Robotic systems, methods, and end-effectors for harvesting produce

ABSTRACT

Robotic systems and specialized end-effectors provide for automated harvesting of produce such as fresh market apples. An underactuated design using tendons and flexure joints with passive compliance increases robustness to position error, overcoming a significant limitation of previous fruit harvesting end-effectors. Some devices use open-loop control, provide a shape-adaptive grasp, and produce contact forces similar to those used during optimal hand picking patterns. Other benefits include relatively low weight, low cost, and simplicity.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/050,048, filed Sep. 12, 2014, the complete contentsof which are herein incorporated by reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under Contract No.2013-67021-20942 awarded by the United States Department ofAgriculture-National Institute of Food and Agriculture (USDA-NIFA)through the National Robotics Initiative (NRI). The government hascertain rights in the invention.

FIELD OF THE INVENTION

The invention generally relates to robotic produce harvesting and, moreparticularly, end effectors and supporting systems for roboticallyharvesting apples and other produce while preferably maintaining harvestquality sufficient for fresh market sales.

BACKGROUND

In the U.S. Pacific Northwest, a large, seasonal labor-force is requiredfor the production of tree fruit crops like fresh market apples,cherries, and pears. The most time and labor-intensive task in fruitcrop production is harvesting. In Washington State alone the apple andpear harvest requires the employment of approximately 30,000 additionalworkers with an estimated harvest cost of $1,100 to $2,100 USD per acreper year. To reduce harvesting costs and dependence on seasonal labor,researchers have developed shake-and-catch systems for the massharvesting of fruits such as berries, cherries, and citrus. Thesetechniques, which apply vibration to the trunk or branch of the tree inorder to separate the fruit, are typically used to harvest fruitdestined for the processing market where there are establishedtolerances for fruit bruising and external defects. There have been someattempts to develop mass harvesting systems for fresh market citrus,cherries, and apples, but the systems demonstrated marginal rates offruit detachment, were only efficient with compatible tree-trainingsystems, or frequently harvested fruit without stems.

The use of robotics technology is another approach researchers havetried for the harvesting of tree fruit. For economic reasons related tochanging labor conditions, scientists and engineers started to activelywork on research and development of fruit-picking robots in the 1980s.These earlier research efforts defined the basic functional requirementsof a fruit-picking robot as the following: i) locate the fruit on thetree in 3D dimensions; ii) approach and reach for the fruit; and iii)detach an undamaged fruit from the tree and deposit it in a container.In order for a fruit-picking robotic system to be commercially viable,it has to be economically feasible and provide harvesting rates (e.g.fruit/second) comparable to those obtained through manual harvesting.Additionally, the system should minimize damage to both the plant andthe harvested fruit to a tolerable level. Despite numerous attempts totransfer industrial robotic technology directly to field based,biologically driven environments, the mechanization of specialty cropharvesting has achieved only limited success primarily due to inadequateaccuracy, speed, and robustness.

Fruit in a single crop possess a high level of variability. For example,tree fruit vary in position, shape, size, and growing orientation. Evenfor the same apple cultivar, parameters such as size and stem lengthvary widely within a single tree. There also exists a year-to-yearvariability in these parameters. Fruit removal technique is usually thelargest cause of fruit injury. Insufficient automated devices exist forfruit harvesting which are able to accommodate these requirements.

Because of rising labor costs, a high workplace injury rate due toladder use, and increasing uncertainty about the availability of farmlabor, the lack of mechanical harvesting is a critical problem receivingmuch attention from both federal agencies (e.g., United StatesDepartment of Agriculture) and state and local organizations (e.g.,Washington Tree Fruit Research Commission).

The basic functional requirements of an apple picking end-effector areto approach and reach for the fruit and then detach an unblemished applefrom the tree. In addition to being efficient, productive andeconomically feasible, it is important that the system not damage thepicked fruit, adjacent fruit, or the tree. The end-effector can damagethe apple by applying excessive force during picking or by employinginappropriate stem separation techniques. Some different techniques havebeen investigated for end-effector designs.

Bulanon and Kataoka (Bulanon & Kataoka, 2010) designed an end-effectorthat used a peduncle holder to apply pressure against the pedunclebefore removing the fruit with a lifting and twisting motion. Thoughthis technique minimized damage to the fruit, the system was constrainedin that the end-effector had to approach the apple horizontally.

Baeten et al. (Baeten, Donne, Boedrij, Beckers, & Claesen, 2008)developed a novel gripper consisting of a flexible silicon funnel thatused vacuum suction to activate the gripping function. During fieldtests the average harvesting time was approximately nine seconds, butstem pulls occurred with approximately 30% of the harvested apples. Itwas also important to sequence apple selection so that adjoining applesin a cluster would not interfere with the picking process.

Zhao et al. (Zhao, Lu, Ji, Zhang, & Chen, 2011) proposed a cuttingend-effector utilizing multiple sensors that demonstrated impressivefruit detachment rates during field tests. Although cutting minimizesthe likelihood of stem pulls, it usually requires more complex controlrequirements, which can lead to higher costs.

SUMMARY

According to an aspect of the invention, solutions are provided toaddress one or more of the following objectives:

-   -   1. Achieve average harvesting time, which is defined as the time        from fruit localization to storage in a container, of six        seconds or less.    -   2. Replicate the kinematics and dynamics of human picking to        minimize fruit damage and stem pulls.    -   3. Minimize the volume of the end-effector workspace in order to        reduce the likelihood of collisions with adjacent fruit and        branches.    -   4. A system not constrained to approaching the fruit from a        single direction.    -   5. Adaptable to harvesting of multiple apple cultivars with        variable, distinct geometries.    -   6. Lightweight, simple, low-cost, and robust to an agricultural        environment.

According to an aspect of the invention, underactuated end-effectors arepresented that are made for the robotic harvesting of produce,especially tree fruit such as apples. The device is optimized for speed,low complexity, suitability for a highly variable field environment, andthe replication of manual hand picking so as to minimize fruit damage.In some embodiments, the end-effector produces a spherical power graspwith a normal force distribution and picking sequence replicatingselected human patterns.

According to another aspect of the invention, an underactuated,tendon-driven end-effector with compliant flexure joints is provided toimprove system performance in the presence of position errors as well asenhance robustness to variable fruit size, shape, and orientation. Insome embodiments, the end-effector has few or no sensors (e.g., fordetecting angular positions or points of contact). An exemplaryend-effector incorporates open-loop control to reduce complexity andimprove picking speed. Examples are presented which determine the normalforces developed during grasping of the apple. Results indicate thatopen-loop, feedforward control can be used to produce optimal normalforce patterns.

Whereas advanced robotic hands designed to perform manipulation taskswith high dexterity are quite complex, underactuated hands are muchsimpler devices that can perform a human-like grasp compliant to theobject without requiring independent actuation of each joint. Anunderactuated device is beneficial because only grasping of the fruit isneeded—the end-effector does not require the capability to performdexterous manipulation. A tendon-driven device gives a transmissionsystem that is lightweight, relatively simple, and enables an adaptivegrasp of multiple apple cultivars.

According to another aspect of the invention, a robotic systemintegrates a manipulator, end-effector, and machine vision system duringfruit harvesting. The manipulator may be a serial link manipulator, forexample. The manipulator and end-effector have a low-cost design. Thesequence of picking motions used by the robot replicate the dynamics ofhuman apple picking.

According to another aspect of the invention, a method of autonomousrobotic harvesting of fruit is described which includes approaching apiece of fruit that is nearest to and reachable by a roboticend-effector, the approach being made along an azimuth angle thatprovides a direct approach; grasping the piece of fruit with a first setof fingers (i.e., primary fingers, described below) and, separately,grasping the stem of the piece of fruit with a second set of fingers(i.e., secondary fingers, described below); and picking and dropping thepiece of fruit. The first set of fingers use, for example, a powergrasp. The second set of fingers use, for example, a pinch grasp.Significantly, to reduce or eliminate bruising of the fruit and/ordamaging the tree branch, it is advantageous to rotate the piece offruit (e.g., through a pendulum motion). In some embodiments, this isperformed simultaneously with retracting the robotic end-effector awayfrom the branch to which the piece of fruit is attached. The rotationbends the fruit stem, causing shear forces, which allow the stem tobreak away from the tree branch without damage to the branch or thefruit.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a robotic harvesting system.

FIG. 2 is a manipulator and end-effector of a robotic harvesting system.

FIG. 3 is a robotic end-effector with the end-effector in an openconfiguration.

FIG. 4 is a top view of the robotic end-effector of FIG. 3.

FIG. 5 is sectional view of a primary finger of the end-effector in FIG.3.

FIGS. 6A and 6B show, respectively, an example optimal graspconfiguration with an end-effector and a grasp configuration used duringmanual apple picking.

FIG. 7 is the robotic end-effector of FIG. 3 with the end-effector in aclosed configuration grasping an object.

FIG. 8 shows the kinematic structure of an exemplary manipulator.

FIG. 9 is a method for harvesting produce with a robotic harvestingsystem.

FIG. 10 is a process for harvesting from a tree with a roboticharvesting system.

FIG. 11 shows a simulated example grasp configuration from a Matlabsolver used to determine angles of link rotation and points of normalcontact for various fruit positions.

FIG. 12 shows the normalized proximal force that develops atstatic-equilibrium for a single finger. The x-y coordinate representsthe position of the center of a circle with a radius of 40 mm.

FIG. 13 shows proximal and distal normal forces compared at fivedifferent actuator loads for each finger. The data points represent themean values of three different iterations and the error bars presentstandard deviation.

FIG. 14 shows the proximal link normal forces that develop during apower grasp of a sphere with a diameter of 80 mm. The normal forces areplotted versus the actuator load.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of a robotic harvesting system 100 for use inorchards, for example. In some embodiments, the robotic harvestingsystem 100 is for apple harvesting. The primary moving componentsinclude an end-effector 101, for grasping apples and removing them fromthe tree, and a manipulator 102, for moving the end-effector as a wholethrough space to reach different parts of a tree. In addition, anexemplary system 100 includes a machine vision system 103 fordetermining target locations of apples to which the manipulator 102 canthen move the end-effector 101. The system 100 also generally includessupport elements such as a storage container 104 in which harvestedapples are collected, a power source 105, a locomotion system 106 formoving the entire system 100 from tree to tree, and other supportelements generally shown by block 107, including, for example, a controlinterface, wireless antenna for remote control, and the like.

FIG. 2 shows an example manipulator 102 and end-effector 101. Themanipulator 102 has a kinematic framework flexible enough to accommodatethe crop environment. Agriculture fields and orchards present anenvironment that is considerably more unstructured than other areas ofautomation like manufacturing plants. Environmental factors includevariable outdoor conditions, complex plant structures, inconsistency inproduct shape and size, and delicate products. With respect to applepicking, a primary source of variation with which the manipulator isconfigured to accommodate is the highly irregular and unstructured appletree.

To improve obstacle avoidance (e.g., branches, tree trunks, other piecesof fruit) during harvesting, a manipulator 102 preferably has sixdegrees of freedom (DOF). In some exemplary embodiments, the manipulator102 is an open chain, serial link manipulator with revolute joints. Thisconfiguration is one that advantageously offers a spherical workspace.Different embodiments may have different maximum reach. For example, asuitable reach is at least 0.5 meters (e.g., 0.5, 0.6, 0.7, 0.8, 0.9, or1.0 meters). The manipulator 102 includes actuators 202 such as modularDynamixel Pro actuators (Robotic Inc., Irvine, Calif.). The frame 203can be fabricated from, for example, aluminum sheetmetal, which islightweight, relatively inexpensive, and sufficiently durable for fruitorchard conditions.

FIGS. 3 and 4 show enlarged depictions of an end-effector 101. Anend-effector 101 is a tendon-driven, underactuated gripping device. Anindividual end-effector 101 includes a plurality of fingers (alsoreferred to interchangeably as “digits”). The exemplary embodimentillustrated in the figures includes three primary fingers 302 each withtwo links (or phalanges) connected by flexure joints. The distal link303 connects to the proximal link 304 with a distal flexure joint 305,and the proximal link 304 additionally connects to the palm 306 with aproximal flexure joint 307. Underactuation and the passive complianceprovided by the flexure joints provide several advantages in theunstructured orchard environment. For example, underactuation betweenthe finger links 303 and 304 helps to ensure a shape-adaptive grasp offruits with variable shapes, sizes, and orientations. The passivecompliance of the flexure joints 305 and 307 also increases robustnessto positioning errors. In the case of unintended collisions which areexpected during harvesting, the flexure joints 305 and 307 can sustainout-of-plane deflection and large deflections without damage.

The palm 306 is defined as a base to which to the fingers 302 areattached. In some embodiments, the palm serves to fix the proximal endof each primary finger 302 to a specific location with respect to theproximal ends of the other primary fingers 302. According to anexemplary configuration, the primary fingers 302 are arrangedsymmetrically around the palm 306. For example, for an end-effector 101with three primary fingers 302, each primary finger is spaced apart fromthe other two primary fingers by 60 degrees (e.g., as measured using thecenter of palm 306 as a center point). Alternative embodiments may havemore than three primary fingers (e.g., 4 or 5 primary fingers); however,three primary fingers are preferred for providing satisfactoryperformance without undue cost and complexity. The primary fingers 302are spaced such that when grasping a sphere of average apple diameter(e.g., 80 mm), the proximal links 304 make contact with the fruit on itsequator. Alternative embodiments intended for harvesting produce ofother sizes (e.g., tomatoes, lemons, grapefruits, oranges, etc.) may beconstructed with a similar configuration to end-effector 101 but withcomponent sizes scaled larger or smaller based on the intended produceto be harvested. An exemplary length of a single primary finger 302 isroughly equivalent to the length of an adult male human index finger.Each link of each finger includes a padding 308 (e.g., a soft rubberpad) for contacting the piece of fruit. The paddings 308 increasefriction and tangential forces between the end-effector 101 and fruitsurface.

FIG. 5 shows an enlarged sectional view of a primary finger 302 based ona bisection along the finger's plane of symmetry. The primary finger 302is illustrated in an open configuration. In the open configuration, anon-zero angle exists between the proximal and distal links. In each ofthe primary fingers 302 a tendon 309 is attached (e.g., tied) at thedistal tip. Leading away from the distal tip, the tendon 309 follows ahollow channel through the distal link 303 and then through the proximallink 304. The tendon 309 is routed in such a way the friction isminimalized. For example, the tendon 309 may be routed over two smalldowel pins 502 in the proximal link 304 in order to reduce friction.There is a single, free-spinning pulley 503 at the base of each primaryfinger 302 that guides the tendon 309 to its termination point on a discdifferential 310 where the tendon is secured (e.g., with a nut). Thedisc differential 310 is attached with a tendon to a horn of theactuator 313. In a preferred embodiment, all of the primary fingers 302are controlled with the single actuator 313. Actuator 313 together withthe disc differential 310 actuates all of the primary fingers 302together. As an alternative, one or more additional actuators may beprovided to allow independent actuation of one or a subset of theprimary fingers 302. However, this generally adds undesirable cost andcomplexity without meaningful gains in overall functionality andperformance of the end-effector 101.

Underactuation of the end-effector 101 is supported by the discdifferential 310. The disc differential 310 is a variant of a seesawmechanism and is essentially a circular disc made of, for example, athin plastic. Each of the three tendons 309 is secured at the edge ofthe disc differential 310 of the differential, e.g., with a nut. Withthe primary fingers 302 are configured symmetrically around the palm306, the termination points of the tendons are arranged symmetricallyaround the disc 310. In the event that one primary finger 302 contactsthe fruit before the other primary fingers 302, the disc differential310 rotates and enables further displacement of the two tendons of theremaining two primary fingers 302 that have not yet made contact withthe piece of fruit.

In some embodiments, an end-effector 101 includes one or more (e.g., atleast two) secondary fingers 311 for applying pressure against the stemduring fruit detachment. The secondary fingers 311 are controlled by aseparate actuator 314 from the primary fingers. In contrast to theprimary fingers 302, the secondary fingers 311 preferably each have onlya single link. Each secondary finger link includes a padding 308 (e.g.,a soft rubber pad) for making contact with the fruit stem. In exampleembodiments, the secondary fingers 311 do not make contact with thefruit. A pair of secondary fingers actuate in a direction of one anotherbut not in a direction of the palm's center. In an open position, anon-zero angle formed between the secondary fingers (30 degrees, forexample, as illustrated in FIGS. 3 and 4). The secondary fingers 311 arein the open position when the manipulator 102 is positioning theend-effector 101 around a piece of fruit. The secondary fingers 311switch to the closed position in parallel with the closing of theprimary fingers 302. After the secondary fingers 311 are moved into theclosed position they are in contact with and put pressure on the fruitstem. The change from the open to closed configurations of a pair ofsecondary fingers 311 is characterizable as a pinch grasp, not unlikethe motion of a human pinch using the forefinger and thumb (particularlywhen the forefinger and thumb are each kept straight). A floating pulley316 provides underactuation between the secondary fingers 311, each ofwhich is connected to the palm 306 with a flexure joint 315. Eachsecondary finger 311 has a single tendon 312. The tendon 312 in eachsecondary finger 311 is secured at the link tips and passes over afloating pulley. The floating pulley 316 is attached with a tendon to ahorn of the actuator 314. In an example embodiment, the actuator 314 isthe only actuator provided for actuating all of the secondary fingers311.

In some embodiments, the palm 306 is circular and may also be concave. Afeature of the palm 306 is the provision of a padding 401 (e.g., a softrubber insert) arranged thereon. The padding 401 serves as an allowablepoint of contact during a grasping task. The manipulator 102 moves theend-effector 101 to a piece of fruit (e.g., an apple) with an openconfiguration. Ideally, the end-effector is switched to the closedconfiguration after contact is made between the padding 401 of the palm306 and the piece of fruit. This helps ensure the piece of fruit iscentrally positioned among the primary fingers 302. In some embodiments,the palm 306 may include a pressure sensor 402 (shown schematically inFIG. 4) for detecting when contact is made between the padding 401 andthe piece of fruit. In other embodiments, a grasp does not requirecontact between the palm 306 and the piece of fruit. The grasp can beperformed and the detachment of the fruit completed without palmcontact.

An exemplary robotic system 100 meets the following five performancecriteria: i) detachment success of at least 90% (detachment success isdefined as the number of successfully harvested ripe fruit per totalnumber of localized ripe fruit present in the workspace of a manipulator102); ii) picking time of 6 seconds or less (picking time is defined asthe time required to pick and store one piece of fruit excluding thetime required for ripeness determination and fruit localization); iii)damage rate to the fruit at 10% or less; iv) usable to harvest multipleapple cultivars; and v) relatively lightweight, simple, and costeffective.

An end-effector 101 accounts for the variability from one piece of fruitto the next. The configuration of the end-effector 101 takes intoconsideration the fruit's physical properties such as friction,firmness, and tensile strength of the stem. Performance of anend-effector 101 can be measured according to the above-identifiedcriteria.

According to some embodiments, an end-effector 101 replicates themanipulation methods of the human hand during apple picking andaddresses the constraints imposed by fruit growth habits. An optimalpattern of fruit removal with the human hand is to grasp the piece offruit (e.g., a single apple) with the thumb and middle finger atopposite points on the equator, place the forefinger against the base ofthe stem, and rotate the fruit against the orientation of the stem.Compared to pulling the fruit away from the tree, this method requiredless force to break the stem-abscission joint and reduced the likelihoodof fruit damage. Though the human grasp is a usually a fingertip grasp,an exemplary end-effector 101 provides an enveloping power grasp of thefruit. The power grasp may be characterized as being spherical. A powergrasp advantageously enhances robustness to position error caused by themachine vision system 103, for example. In a fingertip grasp, contact ismade between the gripping element (e.g., a human hand or roboticend-effector) and the gripped element (e.g., a piece of fruit, an apple)only at the most distal link or phalange, and often more specificallythe most distal tip thereof. In contrast, a power grasp includes contactbetween the gripping element and the gripped element at multiple linksor phalanges of each of one or more fingers (e.g., some or all of thefingers have multiple contact points with the gripped element; each linkor phalange has at least one contact point).

FIG. 6A shows an example of an optimal robotic pattern of grasping anobject such as an apple. The pattern illustrated is an exampleconfiguration optimized for an end-effector with three primary fingersas studied during field experiments. This pattern minimized graspingnormal forces and reduced bruising compared to other tested patterns.Note that this example pattern does not include force being appliedagainst the stem. For comparison, FIG. 6B shows a typical grasp patternused by human laborers during manual apple picking. The harvestingend-effector is the only system component that makes physical contactwith the fruit. In order to minimize damage and improve detachmentsuccess rates, some embodiments include a design based on apple growthhabits and the human hand's manipulation methods during fruit picking.Based on qualitative observations, the typical pattern used byprofessional apple pickers is a two-finger grasp with opposingfingertips placed along the fruit's equator. To separate the fruit fromthe branch, the hand moves the fruit in a pendulum motion while theindex finger pushes the stem, as illustrated by FIG. 6B. Using forcesensors, a field experiment was performed to track and quantify normalforces during three other grasp configurations achievable by anend-effector with three opposing primary fingers. The best graspconfiguration of the three tested is illustrated in FIG. 6A. None of thethree grasps included the application of pressure against the stem.Results from these configurations were then compared with the manualpicking pattern. Pulling while simultaneously rotating the fruit (e.g.,see arrow 601 in FIG. 6A), and thereby bending the stem, produces acombined pulling and pendulum motion. This induces shear forces in thestem. In summary, experimental results show that the application of onlya pulling force along the axis of the stem produces a purely tensileforce in the stem and that the normal grasping forces required to detachthe fruit usually exceed bruising thresholds. To break thestem-abscission joint with lower grasping forces that minimize bruising,bending the stem is generally required so as to induce shear forces.

FIG. 7 shows an end-effector 101 grasping a substantially sphericalobject 702 such as an apple. In some embodiments the grasp of theend-effector 101 is characterizable as a passive, adaptive grasp. Anadaptive grasp describes the process whereby the fingers conform to theshape of the object being grasped. For the end-effector 101 this processis passive because shape conformity during grasping does not requireadditional actuating components that provide energy to the system.Dexterous manipulation of the fruit is not required in some embodiments.Embodiments without dexterous manipulation may provide a normal forcedistribution replicating human patterns at the proximal point ofcontact.

Visual sensing is an essential and primary task for an autonomousrobotic harvesting system. However, vision is often considered abottleneck for developing commercially applicable robotic harvestingsystems. Variable lighting conditions, fruit clustering, and occlusionare some of the significant challenges that limit the performance of themachine vision system in an orchard environment. Prior to harvesting,the robotic system needs to identify and accurately locate the fruit.According to an example configuration for a robotic harvesting system100 (FIG. 1), the machine vision algorithm developed by Silwal et al.(Silwal, A., Gongal, A., Karkee, M., 2014. Apple Identification in FieldEnvironment with Over-the-Row Machine Vision System. AgriculturalEngineering International: Agric Eng Int'l (CIGR Journal), 16(4): 66-75)is used by the machine vision system 103 to identify the apples. Inbrief, this algorithm uses Circular Hough transformation (CHT) toidentify clearly visible fruit, as well as individual apples inclusters, and blob analysis (BA) to identify partially visible fruit. Ithas been previously tested in an orchard environment with 90% fruitidentification accuracy. The physical hardware of the machine visionssystem 103 includes a global camera system which comprises, for example,a single CCD (Charged Couple Device) color camera (Prosilica GC1290C,Allied Vision Technologies, Exton, Pa.) mounted on top of atime-of-flight based three-dimensional (3D) camera (Camcube 3.0, PMDTechnologies, Sigen, Germany). The purpose of this configuration is toacquire color images with the CCD camera to identify the apples and thenobtain their 3D coordinates from the PMD camera to localize theirposition in space. Unlike other fruit harvesting vision systems thatattach a camera to the manipulator or end-effector, the use of a singleset of global cameras doesn't require computationally expensive visualservoing techniques that may constrain manipulation speeds. The machinevision system 103 is required only once, namely at the beginning of eachharvest cycle to identify and localize the apples, thereby saving timeand increasing the efficiency of the entire harvesting system 100.

FIG. 8 and Table 1 show the kinematic structure of a manipulator 102according to an example embodiment. The figure and table also show thekinematic structure's associated geometric parameters. The recursionformulas developed by Wang and Ravani (L. T. Wang and B. Ravani,“Recursive Computations of Kinematic and Dynamic Equations forMechanical Manipulators,” IEEE Journal of Robotics and Automation, Vols.RA-1, no. 3, pp. 124-131, 1985) are used for forward kinematicscomputations. Numerical solutions to the inverse kinematics problem aredetermined with the combined optimization method first proposed by Wangand Chen (L. T. Wang and C. C. Chen, “A Combined Optimization Method forSolving the Inverse Kinematics Problem of Mechanical Manipulators,” IEEETransactions on Robotics and Automation, vol. 7, no. 4, pp. 489-499,August 1991). This algorithm was selected because it has been shown tobe computationally efficient and does not require matrix inversion. Theconvergence tolerance was set at 1E-6, and the joint limits of themanipulator were used as boundary constraints. The inverse kinematicsalgorithm has been developed in Matlab (Mathworks Inc., Natick, Mass.),compiled into a C++ shared library, and integrated with themanipulator's controller in the Microsoft Visual Studio developmentenvironment. In some embodiments, a manipulator's planned trajectory isexecuted using a simple, open loop look-and-move approach. A manipulator102 is preferably configured for use with traditional orchards (e.g.,with naturally shaped apple trees) as well as simple, narrow,accessible, and productive (SNAP) canopies. A SNAP canopy is atwo-dimensional, planer canopy supported by a wire and posts trellissystem whereby most of the branches and fruit are visible and accessibleto machines.

TABLE 1 Kinematic configuration of the 6-DOF manipulator. Link LengthTwist Angle Offset Length Rotation Angle Joint a (m) α (degrees) d (m) θ(degrees) 1 0 90 .177 θ₁ 2 0 90 0 θ₂ 3 0 90 .270 θ₃ 4 0 90 0 θ₄ 5 0 90.219 θ₅ 6 .186 90 0 θ₆

Multiple options for the fabrication of a robotics system andend-effectors according to the invention exist and will occur to thoseof skill in the art based on the teachings herein. For small scaleproduction purposes, end-effector components can be manufactured with 3Dprinting technology (see Example 2 below).

FIG. 9 is a flowchart for a method 900 for harvesting produce (e.g.,apples) with a system (e.g., system 100) and end-effector (e.g.,end-effector 101) according to the invention. The method 900 is anexample for picking an individual piece of produce and is repeatediteratively for picking multiple pieces of produce from a single tree.

The machine vision system first determines a location of a piece ofproduce to be picked (block 901). This location is supplied to themanipulator which guides the end-effector to this location (block 902).The end-effector is maintained in an open configuration while beingmoved into position around the piece of produce. The end-effectorpositioning is terminated after the produce is centrally positionedwithin the end-effector (block 903). This may include the palm of theend-effector making contact with the piece of produce. In some systems apressure sensor in the palm of the end-effector determines when thiscontact is made. At this point the primary fingers of the end-effectorclose about the piece of produce (block 904) and, for a system thatincludes secondary fingers, these are closed to grip and apply pressureto the stem of the produce (block 905). The manipulator then pulls theend-effector away from the initial target location to remove the graspedproduce from the plant (block 906). For apples, for example, this actionmay be a rotation and retraction from the initial produce location. Themanipulator then moves the end-effector to the storage container wherethe piece of produce is released and stored (block 907). The procedure900 is then repeated for another piece of produce.

FIG. 10 shows a flow diagram that is an example process 1000 forharvesting/picking a single tree in an orchard with an end-effector 101.The end-effector balances optimal picking dynamics with simplicity andreduced cost. After the machine vision system 103 localizes the positionof every apple within its field of view (block 1001), the manipulator102 guides the end-effector to an approach point a fixed distance (e.g.,10 cm) away from the nearest fruit (block 1003) in conjunction with amotion planning algorithm (block 1002). When multiple apples are inclose proximity (e.g., in a cluster), the system 103 directs theend-effector to the closest remaining apple first (block 1004). Adecision is made whether the apple is in a reachable workspace (block1005). If not, then the system finds the next nearest apple. Ifreachable, then, following a specific azimuth angle, the end-effector101 makes a horizontal approach along a direct path to the fruit'sposition (block 1006). The azimuth angle is the angle that themanipulator must rotate horizontally for the end-effector to approachthe fruit along a straight line. So, for a fruit located directly infront of the manipulator the azimuth angle is zero. When theend-effector reaches the predetermined fruit position, the primaryfingers and secondary fingers simultaneously close around the fruit andstem, respectively (block 1007). After a pause (e.g., approximately onesecond), the end-effector is then rotated through a predetermined angle(e.g., 30 degrees, counterclockwise) and retracted along the sameapproach path to a fixed distance (e.g., 13 cm) away from the tree. Theend-effector is then opened and the fruit is dropped into a storagecontainer (block 1008). The manipulator then moves the end-effector tothe next fruit approach point and repeats the picking process until allfruit are removed from the tree (decision block 1009).

Example 1

To reduce design complexity and enhance speed of harvesting, an exampleend-effector is provided which has no pressure sensors and utilizesopen-loop, feedforward control. An environmental model is provided forcharacterizing the actuation torque required to produce the desired linknormal forces. Each end-effector finger is a single-acting cable-drivensystem with two links and two flexure joints. The flexures are modeledas simple pin joints with rotational stiffness. For this example, whichdoes not use sensors to detect angular positions or points of contact,modeling the flexures as pin joints with torsion springs is sufficient.However, if desired, a more accurate model of flexure bending is givenOdhner and Dollar (L. U. Odhner and A. M. Dollar, “The Smooth CurvatureModel: An Efficient Representation of Euler-Bernoulli Flexures as RobotJoints,” IEEE Transactions on Robotics, vol. 28, no. 4, pp. 761-772,2012) and can be used to estimate beam deflection. Rotation of theproximal and distal links is coupled until the proximal link makescontact with the object at which point the distal link will continue toindependently rotate until the system is constrained. For thiscable-driven finger the kinematics of the links are coupled and may beexpressed as

r _(a)Δθ_(a) =J _(a)Δθ  (1)

where r_(a) is the radius of the actuator pulley, Δθ_(a) is the angulardisplacement of the actuator pulley, Δθ represents the configurationchange of the links, and J_(a)=(r₁ r₂) is the actuator Jacobian of thefinger. The pulley radii are represented by r₁ and r₂. The quasi-staticequation of equilibrium, which can be found with analytical mechanicsand the principle of virtual work, is

KΔθ+J _(c) ^(T) f _(e) +J _(a) ^(T) f _(a)=0  (2)

where

$K = \begin{pmatrix}K_{1} & 0 \\0 & K_{2}\end{pmatrix}$

represents the joint stiffness of the flexures, J_(c)ε

^(2×2) is the contact Jacobian that maps between contact forces actingon the finger and the joint torques, f_(e)=(_(f) ₂ ^(f) ¹ ) representsthe normal contact force on the proximal and distal links, and f_(a) isthe actuator force. For this two-link mechanism the contact JacobianJ_(c) is

$\begin{matrix}{J_{c} = \begin{pmatrix}b_{1} & 0 \\{b_{2} + {l_{1}\cos \; \theta_{2}}} & b_{2}\end{pmatrix}} & (3)\end{matrix}$

where b₁ is the proximal force location, b₂ is the distal forcelocation, and l₁ is the proximal link length. The normal forces can thenbe found by

f _(e) =J _(c) ^(−T)(−KΔθ−J _(a) ^(T) f _(a))  (4)

In most grasps the proximal normal force will be located on or about thefruit's equator.

The actuation force that produces the normal forces of human pickingpatterns is determined. In reality, this is a complicated processbecause the normal forces are highly dependent upon the final kinematicconfiguration of the underactuated finger. During harvesting operationsposition errors and variation in fruit shape and size will lead tonumerous end-effector grasp configurations. Likewise, in someconfigurations negative normal forces may develop, which indicates lossof contact of the respective link. The actuation force input providedwith open-loop control should ensure that in all possible configurationsthe final grasp does not damage the fruit. A Matlab simulation(Mathworks Inc., Natick, Mass.) was conducted to examine the effect ofposition error on the proximal normal force. The center of a circle withdiameter of 80 mm was placed at different positions in the x-y planehaving an error ranging from −10 to +10 mm in the x-direction and 0 to10 mm in the y-direction. The grid included a total of 400 differentfruit position points. At each point it was assumed that the fruit wasconstrained by the stein/branch system. The simulation used a solver toestimate the configuration where the finger links were tangent to thecircle and then determined the change in link positions Δθ and thepoints of contact b₁ and b₂ assuming a single point of contact on eachlink. An example of an equilibrium grasp configuration calculated by thesolver is shown in FIG. 11. The resulting proximal normal force was thendetermined with eqn. (4). Tangential forces due to friction were notconsidered in this example. The presence of tangential forces shouldincrease the pullout force required to remove the fruit from theend-effector's grasp. The results of the simulation are shown in FIG.12. The force data, which is scaled with unity-based normalization,shows that for a constant actuation force the proximal normal force isgreater for increasing Δθ₁. As shown in FIG. 10, the x-position of thefinger base is located at −0.04 m. For increasing values of x the fruitmoves away from the finger base and the proximal link must rotatefurther in order to contact the fruit. By superimposing force data fromtwo planar fingers in an opposed grasp, it is possible to create vectorfields for the total resultant force on an object. By superimposing thedata from each of the three fingers the greatest resultant force can beexpected near the center of the end-effector at a small distance awayfrom the palm.

Example 2

An example end-effector was used to characterize the relationshipbetween actuation force and normal contact forces. Normal forces weremeasured during a grasp of a plastic sphere with radius of 40 mm. Thesphere was located symmetrically with respect to the end-effector,meaning the centerline of the end-effector was coincident with thecenter of the sphere resting on the palm. Three piezoresistive forcesensors (Tekscan Inc., Boston, Mass.) were attached to the contactlocations on the proximal links. To complete a power grasp theend-effector's actuator, which is nonbackdriveable, was operated intorque mode and driven to its stall point. The normal force was thenmeasured at static equilibrium. This measurement was repeated forincreasing actuator loads. The experimental set-up corresponds with theconfiguration illustrated in FIG. 7. The same test was repeated for thedistal normal forces. The results for the normal force measurements atfive different actuator loads are shown in FIG. 13. As expected, forincreasing tendon tension the proximal normal force is significantlyhigher than the distal force. While the change in proximal force isrelatively linear for increasing tendon tension, the change in distalforce is highly irregular and shows sharp jumps interspersed withhorizontal slopes. The proximal links remain static once contact withthe surface is made. The distal links, however, often adjusted theirequilibrium configurations at new actuator loads. FIG. 14 shows acomparison of the proximal normal forces measured for the three fingerssimultaneously at ten different actuator loads. The results present themean values from six different iterations. For this particular graspapproximately 10% of the actuator's maximum torque value was required toproduce proximal normal forces representative of those developed duringmanual picking of apples, which is approximately 7 N. Based on thesimulation results presented in FIG. 12, the contact forces developedwith this actuation torque for asymmetric grasp configurations shouldremain below the 11 N force threshold that caused bruising during fieldtests of hand-picking patterns. Though the normal force distribution wassimilar for each of the fingers, the grasp was not force-isotropic aswould be expected for a symmetric grasp. Because normal forces arehighly configuration dependent, slight variations in the sensorplacement can significantly impact the final results. Also, whenincreasing tendon tension the point of proximal contact sometimesslightly adjusted. The change in position of normal contact relative tothe sensor may be a source of some of the variance between the proximalforce values.

The components of the end-effector in Example 2 were fabricated fromsolid models by a Replicator 2X printer (MakerBot Industries, New York).Additive manufacturing minimizes the fasteners required for assembly andlessens the total weight of the device. The fingers were printed asmonolithic parts of ABS plastic. Molds for the finger pads were includedin the solid parts as thin shells. A soft urethane rubber (Vytaflex 30,Smooth-On, Inc.) was poured into both the finger pad cavities and thepalm. After the elastomers cured, the shells of the finger pad moldswere cut away. Dovetail joints included on the fingers and palm hold therubber pads in place. The flexures for the primary and secondaryfingers' joints were printed with flexible filament (Ninjaflex, FennerDrives, Inc.) and inserted directly into cavities in the links. Multiplehigh-strength low-friction filaments will occur to those of skill in theart for use as the tendons. As an inexpensive option, high strengthfishing line (e.g., 100-lb) was used in the Example. The actuatorpulleys, floating pulley, and differential were also printed parts. Theactuators selected for the primary and secondary fingers are theDynamixel MX-64AR (Robotis Inc., Irvine, Calif.) and MX-28AR,respectively. It is advantageous to employ a servos providing thecapability of position, speed, and torque-based commands. Dynamixelservos include a 4,096 step absolute encoder after gear reduction andintegrate a DC motor, reduction gearhead, PID controller, motor driver,and data network. These particular models have stall torques of 7.3 N-m(MX-64AR) and 2.5 N-m (MX-28AR). Because the operating voltage of theend-effector servos is 12 VDC, it used a separate power supply than themanipulator, which operates at 24 VDC. The servos are controlled in theMicrosoft Visual Studio C++ development environment using the softwaredevelopment kit (SDK) provided by the manufacturer.

The mass of the assembled end-effector in Example 2 was 0.4 kg. Based onthe average mass of the apples harvested, the total payload of theend-effector and harvested fruit is generally less than 0.7 kg. Some keygeometric and physical parameters of the end-effector fingers areprovided in Table 2. The orthogonal distance from the midline of theflexure to the tendon entry point is used for the equivalent pulleyradii. The stiffness of the flexure joints was experimentallydetermined. The joints were modeled as simple torsion springs, and theirrotational stiffness was measured with a load cell. The stiffness ratiok₂/k₁ of two between the joints plays an influential role in the natureof the coupled motion between the links. The arrangement of the fingersis designed to provide a spherical power grasp fully encompassing thefruit. In this grasping sequence the proximal link makes contact withthe object first before the distal link flexes to cage the fruit. Inorder to ensure this two-phased motion, the distal flexure joint isstiffer than the proximal flexure joint. Individual features or multiplefeatures from the Example 2 end-effector may be incorporated intovarious alternative embodiments in the practice of the invention. ThisExample's end-effector is illustrative only and is not necessarilyintended to be limiting.

TABLE 2 Physical and geometric parameters of the end-effector of Example2. Joint Pulley Resting Length Stiffness Radius Angle Link l (m) k(N-m/rad) r (m) θ (°) 1 0.07 0.055 0.00956 45 2 0.042 0.111 0.00716 35

While the descriptions herein have largely referred to the harvest offruit and especially apples, the descriptions presented are ofnon-limiting examples. Embodiments of the invention may be used forharvesting any of a variety of types of produce, be it vegetables orfruit. Exemplary end-effectors according to the teachings herein areespecially well suited for substantially spherical produce such asoranges and grapefruit. Pears, lemons, limes, and other fruit orvegetables may also be harvested with variations of the exampleembodiments disclosed herein.

While exemplary embodiments of the present invention have been disclosedherein, one skilled in the art will recognize that various changes andmodifications may be made without departing from the scope of theinvention as defined by the following claims.

Symbols

l₁, l₂ Length of proximal and distal links

k₁, k₂ Stiffness of proximal and distal flexures

K Stiffness matrix

r₁, r₂ Equivalent pulley radii of proximal and distal joints

r_(a) Actuator pulley radius

Δθ_(a) Angular displacement of the actuator pulley

Δθ Configuration change of the links

J_(a) Actuator Jacobian matrix of the finger

f_(e) Vector of normal contact forces

J_(c) Contact Jacobian matrix that maps between link contact forces andjoint torques

b₁, b₂ Location of proximal and distal normal forces

1-25. (canceled)
 26. A robotic produce harvesting system, comprising: anend-effector comprising a plurality of fingers which are tendon-driven,and a plurality of actuators configured for independent actuation of theplurality of fingers, wherein the end-effector is underactuated, andwherein the end-effector is configured to grasp a piece of produce to beharvested with a power grasp; a manipulator for positioning theend-effector in three-dimensional space; and a machine vision system forproviding a location of the piece of produce to be harvested.
 27. Thesystem of claim 26, wherein the manipulator is configured to executeplanned trajectories using a look-and-move approach.
 28. The system ofclaim 26, wherein the end-effector has a passive, adaptive grasp. 29.The system of claim 26, wherein the end-effector utilizes open-loop,feedforward control.
 30. The system of claim 26, wherein theend-effector is configured to grasp a piece of produce selected fromapples, tomatoes, lemons, grapefruits, and oranges.
 31. The system ofclaim 26, wherein the plurality of fingers comprises three or moreprimary fingers.
 32. The system of claim 31, wherein the end-effectorfurther comprises a palm.
 33. The system of claim 32, wherein theprimary fingers are arranged symmetrically about the palm.
 34. Thesystem of claim 26, wherein the manipulator and end-effector areconfigured to move in a combined pulling and pendulum motion.
 35. Thesystem of claim 26, wherein the machine vision system is configured todetermine a location of a piece of produce that is nearest to andreachable by the end-effector.
 36. The system of claim 26, wherein themachine vision system is configured to identify and localize pieces ofproduce at the beginning of each harvest cycle.
 37. The system of claim36, where the machine vision system is configured to identify andlocalize pieces of produce only at the beginning of each harvest cycle.38. The system of claim 26, wherein the machine vision system isconfigured to identify clearly visible fruit using Circular Houghtransformation.
 39. The system of claim 38, wherein the machine visionsystem is further configured to identify individual apple clusters usingCircular Hough transformation.
 40. The system of claim 26, wherein themachine vision system is configured to identify partially visible fruitusing block analysis.
 41. The system of claim 26, wherein the machinevision system comprises a CCD color camera and a time-of-fight basedthree-dimensional (3D) camera.
 42. The system of claim 41, wherein themachine vision system is configured to identify apples by acquiringcolor images with the CCD camera and to localize apples inthree-dimensional space with the 3D camera.
 43. A method of autonomousrobotic harvesting of produce, comprising determining with a machinevision system a location of a piece of produce to be harvested;positioning with a manipulator an end-effector in three-dimensionalspace such that the end effector is guided to the determined location ofthe piece of produce to be harvested; grasping the piece of produce witha plurality of tendon-driven, underactuated fingers of the end-effector,wherein the grasping comprises a power grasp, and wherein the graspingcomprises independently actuating the plurality of fingers with aplurality of actuators; and picking and dropping the piece of produce.44. The method of claim 43, wherein the positioning step comprisesapproaching the piece of produce along an azimuth angle.